Persistence Ensured QoS 1/2 Message Handling
Introduction
As the Mosquitto Broker is optimized for efficient message processing and high message throughput secure handling of message with QoS 1 or 2 is challenging. From a publisher perspective receiving a PUBACK for a message published with QoS1 or receiving a PUBCOMP for a message send with QoS 2 should guarantee the message is not lost. Especially guarantees for persistence enabled brokers in case of software or hardware crashes for the handling of messages with QoS 1 or 2 needs to be well-defined.
With the 2.9 Mosquitto Pro Broker the default behavior for the Persist SQLite Plugin and for the HA Broker has changed !
The default behavior for some persistence implementations has been changed to ensure zero risk of loosing messages or message states. This may lead to increasing latency for the publisher to receive acknowledge of messages send with QoS 1 or 2. Config switches in the affected persistence implementations to restore the old behavior have been implemented.
Old Behavior until Release 2.8
In general on normal operation of the Mosquitto Broker handling persistence works fine independent of the persistence implementation used. As long as the broker is operated normally and shutdown or restarted using a well-defined shutdown mechanism there is no risk to lose any message or state. Only in case of an unexpected crash or hardware failure there is a risk to lose messages or states.
Eclipse Mosquitto Snapshot Store
With the traditional snapshot persistence store implemented in the Eclipse Mosquitto Broker the persistence state will be snapshoted into a binary file in a fixed time interval. Additionally, the state will be stored in the snapshot file on a shutdown of the broker. Messages sent to the broker with QoS 1 or 2 will be acknowledged be the broker immediately in memory. In case of an unexpected system or hardware crash any persistence information after the last snapshot generation will be lost.
Persist SQLite Plugin
The Persist SQLite Plugin writes any kind of state change to SQLite database. In contrast to the snapshot generation the updates are added to the database continuously. Therefore, the message processing does get blocked for significant amount of time. But to reduce the real I/O load and reduce performance degradation from the database operation the changes will be committed to the database in configurable intervals as well. Messages published with QoS 1 or 2 were added to the current open database transaction and acknowledge to the publisher it was out. In case of an unexpected system or hardware crash any data in the open transaction, which was not committed to the database, will be lost.
High Availability Cluster
In the high availability cluster the active leader processing the message will add the message to the consensus cluster and forward this change to the follower nodes. The acknowledgment will be sent to the publisher immediately. But the communication with the other cluster members and sending the message to the subscribers happens asynchronously decoupled from the main broker processing. If the leader node fails in some way the message may not get send out to the subscribers. And in case the majority of the remaining cluster nodes electing a new leader have not received the latest changes the message may get lost.
Behavior since Release 2.9
Eclipse Mosquitto Snapshot Store
The behavior of the traditional snapshot persistence implemented in the Eclipse Mosquitto Broker is not changed. But due to the snapshot creation blocking the message processing we anyway advice to use a different persistence backend in Mosquitto Pro.
Persist SQLite Plugin
Since this release the acknowledgment of any message sent be a publisher with QoS 1 or 2 will be delayed until the database transaction was committed. The delay a single publisher will experience depends on the Persist SQLite Plugin configuration. With the default setting using a 5-second plugin_opt_flush_period the statistically expected delay is 2.5 seconds. Starting with this release the plugin_opt_flush_period maybe set to 0. With this setting the Persist SQLite Plugin will commit the transaction every processing cycle, if a database change was added in the cycle. This way the delay maybe reduced to some milliseconds. Using the new operation mode the message will always be committed into the database before the acknowledgment is sent out to the publisher. Together with the plugin_opt_sync configuration flag you may configure a Mosquitto Broker to sync the change even to the disk before sending out an acknowledgment to the publisher.
# Use Persist SQLite Plugin with maximum persistence ensured and minimum delay
global_plugin /usr/lib/mosquitto_persist_sqlite.so
plugin_opt_flush_period 0
plugin_opt_sync extra
To restore the old behavior of the Persist SQLite Plugin without waiting on the commit of the change in the database set the new config entry plugin_opt_fast_acknowledge to true:
# Restore old Persist SQLite Plugin with behavior to acknowledge publisher with commit of transaction
global_plugin /usr/lib/mosquitto_persist_sqlite.so
plugin_opt_fast_acknowledge true
High Availability Cluster
Since this release the high availability cluster will wait sending the acknowledgment message to the publisher until the message state is successfully added to the cluster consensus log. This means the message is accepted by a majority of the cluster nodes and can not be lost anymore. In normal cluster operation the publisher may expect some milliseconds of delay between publishing the message and receiving the acknowledgment message from the broker.
To restore the old behavior of the HA broker without waiting on acceptance of the change by a cluster majority set the new config entry plugin_opt_fast_acknowledge to true:
global_plugin /usr/lib/cedalo_mosquitto_ha.so
plugin_opt_fast_acknowledge true
Limitations
Due to the changes a single publisher may experience a significant higher latency for receiving an acknowledgment for a message published with QoS 1 or 2. This does not add any additional limit to the overall message throughput of message published with QoS 1 or 2 from multiple publisher. But depending on the MQTT client implementation it may limit the throughput of message send with QoS 1 or 2 from a single publisher. Assuming the MQTT client implementation does allow 20 messages to be in-flight to the broker and a average delay of 250ms the message throughput of message send with QoS 1 or 2 will be limited to 80 messages per second for this client.
The creation of the snapshot in the Eclipse Mosquitto Broker may take significant amount of time depending on the size of the state information and the performance of I/O system. As the broker will not be able to process any messages during the snapshot generation we advise to use a different persistence backend for the Mosquitto Pro broker anyway.